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Abstract 

The programs within NASA's Aeronautics Research Mission Directorate (ARMD) conduct research and 
development to improve the national air transportation system so that Americans can travel as safely as possible. 
NASA aviation safety systems analysis personnel support various levels of ARMD management in their fulfillment 
of system analysis and technology prioritization as defined in the agency’s program and project requirements. This 
paper provides a framework for the assessment of aviation safety research and technology portfolios that includes 
metrics such as projected impact on current and future safety, technical development risk and implementation risk. 
The paper also contains methods for presenting portfolio analysis and aviation safety Bayesian Belief Network 
(BBN) output results to management using bubble charts and quantitative decision analysis techniques. 
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Introduction 

The programs within the National Aeronautics and Space Administration’s (NASA) Aeronautics Research Mission 
Directorate (ARMD) conduct research and development to improve the national air transportation system so that 
Americans can travel as safely as possible. NASA aviation safety systems analysis personnel support various levels 
of ARMD management in their fulfillment of system analyses and technology prioritizations as defined in the 
agency’s Research and Technology Program and Project Management Requirements (National Aeronautics and 
Space Administration, 2013). Traditionally, multidisciplinary optimization techniques have been used in aerospace 
systems analyses (Alexandrov, Kaplan, Oran & Boris, 2010; Tong, Jones, Arcara & Haller, 2005), but overly 
complex methods can sometimes have pitfalls: 

The proponents of many R&D project- selection methods have not appeared too concerned about 
the level of complexity of their particular technique. Some approaches presented in the literature 
are so mathematically elaborate that they necessitate the assistance of an expert decision analyst 
in order to be usable by most real-world managers. As a consequence, very little use has been 
made by managers of many of these approaches (Henriksen & Traylor, 1999) 

Many organizations external to the NASA ARMD are also developing frameworks and methods for evaluating the 
impact of National Airspace System (NAS) enhancements. Examples include the Joint Implementation 
Measurement and Data Analysis Team’s (JIMDAT) process for evaluating proposed Commercial Aviation Safety 
Team (CAST) system enhancements (CAST, 2004), the Federal Aviation Administration’s (FAA) development of 
models to determine the safety risks in the Next Generation Air Transportation System (Goldner & Borener, 2006) 
and the National Aerospace Faboratory (NFR) of the Netherlands’ aviation system modeling activities (Roelen & 
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Blom, 2013). The metrics and portfolio analysis results presentation methods for aviation safety systems analyses 
used by NASA ARMD are presented in this paper. 

Framework Metrics 

Almost two decades ago, when the White House Commission on Aviation Safety and Security (1997) challenged the 
Federal Aviation Administration (FAA), NASA and industry to reduce the aviation fatal accident rate. In their 
recommendation, the Commission stated the following: 

Cost considerations and mathematical formulas, however, should never be dispositive in making 
policy determinations regarding aviation safety they are one input for decision making. Further, 
non-quantifiable safety and security benefits should be included in the analysis of proposals. 

Both qualitative and quantitative metrics were chosen by the safety system analysis personnel at NASA based on 
experience obtained from conducting analyses for a variety of unpublished internal studies (e.g.. Quick Tall Poles 
Study of Modified Vehicle System Safety Technologies (VSST) Project Technical Challenges in 2011). The 
analyses are conducted using a combination of the following five (5) metrics: (1) Projected Impact on Current Safety 
Tall Poles (Evans, 2014), (2) Projected Impact on Future Safety Tall Poles, (3) Technical Development Risk, (4) 
Implementation Risk and (5) Projected Impact on Aviation Safety Risk Scenarios. 

Projected Impact on Current Safety Tall Poles 

The White House Commission on Aviation Safety and Security recommended that the FAA, NASA and industry 
“should establish a national goal to reduce the aviation fatal accident rate by a factor of five within ten years and 
conduct safety research to support that goal”. The NASA Aviation Safety Program (AvSP) that was created in 1997 
in response to this recommendation used the aviation fatal accident rate as a metric to evaluate the NASA AvSP 
technology portfolio. A more qualitative metric for evaluating current safety was developed as part of the Systems 
Analysis of the Aviation Safety Program (SAoAS) study (Jones, Reveley, Withrow, Evans, Barr & Leone, 2013) 
that was conducted for the Aeronautics Research Mission Directorate in early 2010. The metric. Projected Impact on 
Current Safety Tall Poles is a qualitative assessment of the expected impact of a technology (research outcome, 
technology or safety enhancement) on a Current Safety Tall Pole. The Current Safety Tall Poles that were 
developed in 2010, are based on all recorded accidents and incidents from 1997-2006, involving commercially built 
fixed-wing airplanes operating under Federal Aviation Regulations (FAR) Part 121 (scheduled air carriers). Part 135 
(air charter) or Part 91 (general aviation). An updated version of the Current Safety Tall Poles based on 2001-2010 
data was developed in 2013 (Barr, 2013) and are summarized in Exhibit 1. 

Exhibit 1. Updated Current Safety Tall Poles for FAR Parts 121, 135-S, 135-NS and 91 
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The Projected Impact on Current Safety Tall Poles is defined as the impact that a technology will have on 
reducing the likelihood of a hature occurrence of a particular Current Safety Tall Pole. Also, for the assessments that 
were conducted with this metric, the technologies were assumed to be fully realized. Through a process of consensus 
between aviation safety systems analysis team members, each technology in the research portfolio was evaluated 
against the set of Current Safety Tall Poles. Prior to delivery of final assessment results to upper management, safety 
systems analysis personnel conduct meetings with technical leads and researchers to review accuracy of assumptions 
and preliminary results 

Projected Impact on Future Safety Tall Poles 

Similarly, the Projected Impact on Future Safety Tall Poles is the expected impact of a technology on a Future 
Safety Tall Pole. The rationale for developing this metric is to address safety concerns that may or may not exist in 
the current National Airspace System. Just as texting while driving is a new risk in automobile safety that did not 
exist in historical accident data, changes in aircraft and airspace related technologies (e.g., composite materials) 
impact the future risks of aviation. 

The original set of Future Safety Tall Poles was also created during the 2010 SAoAS study. The Future 
Safety Tall Poles are the most critical future safety risk issues that were identified by multiple experts, external to 
NASA, as areas of concern in terms of aviation safety. The Future Safety Tall Poles have also been updated since 
the SAoAS study to incorporate more recently published reports regarding aviation safety issues. A comparison of 
the 2010 and 2013 versions of the Future Safety Tall Poles is displayed in Exhibit 2. The implementation of the 
Projected Impact on Future Safety Tall Poles metric is identical to process described in the previous section for the 
Current Safety Tall Poles metric. 


Exhibit 2. Comparison of 2010 and 2013 Future Safety Tall Poles. 


2010 

2013 

Runway safety 

Loss of control-in flight 

Increasing complexity and reliance on automation 

Runway safety 

Loss of control-in flight 

Increasing complexity and reliance on automation 

Icing/ice detection 
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Technical Development Risk 

Technical Development Risk (TDR) is a structured means of executing the NASA Technology Assessment (TA) 
process (National Aeronautics and Space Administration, 2007) by examining the potential impediments in 
developing a technology from a baseline Technology Readiness Level (TRL) to an expected TRL. The TDR metric, 
which is based on risk assessment approaches from several sources (Cox, 2000; Abramson & Book, 2000; Volpe 
National Transportation Center, 1997), does not include management risks that are associated with program/project 
resources (e.g., labor, travel), infrastructure, milestones, etc. 

To implement the TDR assessment process, aviation safety systems analysis team members review project 
plans, technology lists and interview project managers, technical leads and researchers to gather information about 
each technology in the project portfolio. Using this information and the criteria shown in Exhibit 3, NASA aviation 
systems analysis personnel determine the TDR ratings for each technology in the portfolio via consensus. 
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Exhibit 3. Technical Development Risk Criteria 


Risk Driver Category 

Low (1) 

Medium (2) 

High (3) 

Required Technical 
Advancement 

Use of existing 
hardware/software or 
minor modifications 

Use of existing hardware with 
major modifications; use of 
existing software with some new 
modules/code development. 

State of the Art or beyond. 

Technology Status 

In use or prototype 
exists 

Under development 

Concept stage 

Complexity 

Technology design 
simple and/or consists 
of few parts 

Technology design moderately 
complex and/or consists of 
multiple parts. 

Technology design highly 
complex and uncertain 
and/or consists of multiple, 
highly integrated parts. 

Dependencies 

Independent of other 
technologies 

Dependent on proven 
technologies, equipment, and/or 
test data. 

Dependent on unproven 
technologies, and/or test 
data. 

TestabilityA/erifi ability 

Technology 
performance can be 
fully tested/verified 
using existing 
data/information 

T est/verification of technology 
performance requires 
development of new 
data/information; all adverse 
conditions can be modeled. 

Technology performance 
cannot be tested/verified 
under all adverse 
conditions. 

Impact on Technology Goal 

nonessential or 
minimum impact on 
technology 
performance 

partial technology performance 
can be obtained or alternatives 
available 

"show stopper" technology 
cannot be developed and 
is infeasible 


Implementation Risk 

The Implementation Risk (IR) assessment is an examination of the possible impediments to actual implementation 
of proposed technologies in the NAS. One of NASA’s strategic objectives is advance aeronautics research to 
“enable a revolutionary transformation” of air travel “for safe and sustainable U.S. and global aviation” (NASA, 
2014). The importance of the IR metric in aviation safety portfolio assessment is that it helps project managers 
identify implementation risk drivers such as market penetration and dependencies (e.g., new training and/or 
infrastructure requirements). 

The IR criteria are based on various assessment techniques used by the FAA and industry (Jones & 
Reveley, 2003). The criteria, shown in Exhibit 4, were applied to the project research portfolio using the same 
process described for the TDR assessment in the previous section. 


Exhibit 4. Implementation Risk Criteria 


Risk Driver 
Category 

Low (1) 

Medium (2) 

High (3) 

Implementation 
Readiness Level 
(IRL) Impacts 

Current certification process 
easily adaptable 

Certification process historically 
difficult and/or rigorous. No FAA 
mandate; advisory only 

Certification may be controversial, 
precedent setting, or untried. 
International rule-making required 

Dependencies 

No new training or infrastructure 
requirements 

Dependent on immature 
technologies. Additional 
automation/IT infrastructure 
required for transfer of NASA 
research and technology (R&T). 

Requires FAA regulation 
modification. Infrastructure builds 
dependent upon or diverse from 
FAA NAS Architecture Study. 

Market 

Penetration 

Business as usual level of 
stakeholder investment 
requirements. FAA mandate with 
retrofit or training subsidization 
program 

Provides product line growth In 
established market. FAA 
mandate without subsidization 

Large stakeholder investment 
requirements. 

Market Impact 

User acceptance high (customer 
pull/shared costs). Airline ops 
impacts minimal. Includes 
transfer of improved processes to 
established programs. Decreased 
direct operating costs (DOC). 

Airline ops impacts. Additional 
training requirements. Initiation 
of applicable programs required 
for transfer of NASA R&T. 
Increased DOC. 

User acceptance low. 
Entrepreneur market. 
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Projected Impact on Aviation Safety Risk Scenarios 

The newest metric added to the aviation safety assessment framework is the Projected Impact on Aviation Safety 
Risk Scenarios. The purpose of this metric is to provide a more quantitative means of evaluating the Safety Tall 
Poles. NASA aviation systems analysis personnel are modeling various aviation safety scenarios using Bayesian 
Belief Networks (BBN) that are based on a combination of historical accident data and subject matter expert 
knowledge. Loss of control (LOG) was selected as the first aviation safety risk scenario to be modeled because 
between 1997 and 2006, LOG accidents comprised 24% of fatal accidents and 54% of total fatalities for commercial 
air carriers (Jones, Reveley, Withrow, Evans, Barr & Leone, 2013). After the loss of control accident framework 
(LOGAF) model (Shih, Ancel, & Jones, 2012) was completed, two other aviation safety risk scenarios are in 
development to address other Safety Tall Poles. The FLight Automation Problems (FLAP) model examines issues 
related to the Future Safety Tall Pole Increasing Gomplexity and Reliance on Automation (Ancel & Shih, 2014). 

The Runway Incursion (RI) model addresses risks involved with runway incursions (Luxhoj, Ancel, Green, Shih, 
Jones & Reveley, 2014) 

The BBN aviation safety risk models are built using a commercial-off-the-shelf (GOTS) software tool that 
contains a sensitivity analysis function. This function can be used to identify the most influential nodes in a model. 
For example, the top causal risks in the LOGAF model are shown in Exhibit 5. 

Exhibit 5. Top Gausal Risks from Sensitivity Analysis of LOGAF Baseline Model 
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All of the nodes in a scenario model are normalized and the Projected Impact on Aviation Safety Risk Scenario, y, is 
calculated as follows: 


m 




Where 


aij = impact of technology i on Scenario causal node j 

Cj = normalized maximum sensitivity value for Scenario causal 
node j 


Technology impact 
on node j 
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+ 1 

None 

0 
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Portfolio Analysis Results Presentation Methods 

A number of methods have been used by the aviation safety systems analysis personnel to display the portfolio 
assessment results that can be generated using the metrics defined in the previous section. In this section of the 
paper, some of the methods most frequently used by the NASA ARMD for aviation safety technology portfolio 
assessments are described. 

Summary Tables 

Simple summary tables of metric results have been used by NASA ARMD for several safety systems analysis 
studies. In Exhibit 6, technologies that were identified as having a potential impact on a Safety Tall Pole are 
indicated by in the table. 

Exhibit 6. Sample Current Part 135 Safety Tall Poles Results Using Checkmark Ratings. 
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Exhibit 7 also uses the Safety Tall Poles metrics, but technologies were rated using values of direct, indirect or no 
impact. The Technical Development Risk and Implementation Risk metrics are evaluated using high, medium and 
low ratings. A table containing sample TDR results is shown in Exhibit 8. 
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Exhibit 7. Sample Current Part 121 Tall Poles Results Using Direct/Indirect Ratings 
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Bubble Charts 

Bubble charts have been used in R&D portfolio analysis (Roussel, Saad, & Erickson, 1991; MacMillan & McGrath, 
2002) to visually examine three or more metrics at one time. In the assessment framework used by the NASA 
aviation safety systems analysis team, each bubble in the chart represents a technology. The size of the bubble 
represents the total safety tall poles tally (i.e., total number of current and safety tall poles impacted by the product). 
The color of the bubble can be changed to help display pertinent information decision makers. For example, in 
Exhibit 9, all of the products in a project’s technology portfolio are color coded based on their total impact on the 
current and future safety tall poles (green = greatest impact; red = least impact). A quick glimpse of the chart reveals 
that the two technologies color coded green (C-2 and C-4) are expected to have the greatest safety impact. Another 
display method used is to color code the bubbles based on the Project Subprojects as shown in Exhibit 10. All of the 
technologies in Subprojects A and B have mostly low implementation risks, but the technologies in Subproject C 
have a wide range of implementation risk. 
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Exhibit 9. Bubbles Colored Coded by Impact on Total Safety Tall Poles 



Exhibit 10. Bubbles Colored Coded by Notional Subproject 



Portfolio Ranking Using Quantitative Decision Analysis Techniques 

Traditional decision analysis methods such as weighted sum models and Analytic Hierarchy Process, have been 
used by NASA ARMD personnel to prioritize technologies in aviation safety portfolios (Exhibit 11). Metrics in the 
portfolio can have equal weights or program and project decision makers can assign weights to the metrics 
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according to their perceived value to their research goals. The weights assigned to each metric should be obtained 
from decision makers prior to implementation of any decision analysis method so that analysis results are not 
manipulated to obtain a desired outcome. 

Exhibit 11 . Sample Portfolio Prioritization Using Weighted Sum Model (All Metrics Equal) 


Product 



■ Technical Development Risk ■ Current Tall Poles 
□ Future Tall Poles ■ Implementation Risk 


Conclusion 

The framework (metrics and presentation methods) presented in this paper has been developed as a result of over ten 
years of implementing analyses of aviation safety technology portfolios. The advantages of this framework include: 
(1) it is not mathematically elaborate and therefore usable by real-world managers, (2) it does not rely solely on fatal 
accident rate for measuring safety and (3) it includes multiple ways to measure non-quantifiable safety benefits. 
Future work will continue in the development of additional models for implementation of the Projected Impact on 
Aviation Safety Risk Scenarios metric. Other metrics pertaining to costs/safety benefits are also expected to be 
included in future version of this framework. 
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